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1 Incorporation By Reference 

2 U. S. Patent Application Serial No. 09/723,754, filed on November 26, 2000, in the names 

3 of Gideon P. Stein, Ofer Mano And Amnon Shashua, and entitled "System And Method For 

4 Estimating Ego-Motion Of A Moving Vehicle Using Successive Images Recorded Along The 

5 Vehicle's Path Of Motion 11 (hereinafter referred to as "the Stein I patent application"), assigned to 

6 the assignee of the present application, incorporated herein by reference. 

7 U. S. Patent Application Serial No. 09/723,755, filed on November 26, 2000, in the names 

8 of Gideon P. Stein And Amnon Shashua, and entitled "System And Method For Generating A Model 

pi 

,j§> Of The Path Of A Roadway From A Sequence Of Images Recorded By A Camera Mounted On A 

W Moving Vehicle" (hereinafter referred to as "the Stein II patent application") assigned to the assignee 

43 of the present application, incorporated herein by reference. 

hi 

i 2 Field Of The Invention 

1*3 The invention relates generally to the field of systems and methods for generating an estimate 

fft as to the structure of a roadway from a vehicle and more specifically to systems and methods for 

*¥ 5 generating an estimate using an image recorded from the vehicle. 

1 6 Background Of The Invention 

17 Accurate estimation of the structure of a roadway ahead of a vehicle is an important 

1 8 component in autonomous driving and computer vision-based driving assistance. Using computer 

1 9 vision techniques to provide assistance while driving, instead of mechanical sensors, allows for the 

20 use of the information that is recorded for use in estimating vehicle movement to also be used in 

21 estimating ego-motion identifying lanes and the like, without the need for calibration between 

22 sensors as would be necessary with mechanical sensors. This reduces cost and maintenance. 
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1 There are several problems in determining the structure of a roadway. Typically, roads have 

2 few feature points, if any. The most obvious features in a road, such as lane markings, are often 

3 difficult to detect and have a generally linear structure, whereas background image structures, such 

4 as those associated with other vehicles, buildings, trees, and the like, will typically have many 

5 feature points. This will make image- or optical-flow-based estimation difficult in practice. In 

6 addition, typically images that are recorded for roadway structure estimation will contain a large 

7 amount of "outlier" information that is either not useful in estimating roadway structure, or that may 

8 result in poor estimation. For example, in estimating of roadway structure, images of objects such 

9 as other vehicles will contribute false information for the road structure estimation. In addition, 
10 conditions that degrade image quality, such as raindrops and glare, will also make accurate road 
IP structure estimation difficult. 

ISP 

'fc Summary Of The Invention 

%J 

f IB The invention provides new and improved systems and methods for generating an estimate 

IjcA of the structure of a roadway using an image recorded from the vehicle. 

ess: 

I $5 In brief summary, the invention provides a road skeleton estimation system for generating 
i|6 an estimate as to a skeleton of at least a portion of a roadway ahead of a vehicle. The road skeleton 

II i 7 estimation system includes an image receiver and a processor. The image receiver is configured to 

18 receive image information relating to at least one image recorded ahead of the vehicle. The 

1 9 processor is configured to process the image information received by the image receiver to generate 

20 an estimate of the skeleton of at least a portion of the roadway ahead of the vehicle. 
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1 Brief Description Of The Drawings 

2 This invention is pointed out with particularity in the appended claims. The above and 

3 further advantages of this invention may be better understood by referring to the following 

4 description taken in conjunction with the accompanying drawings, in which: 

5 FIG. 1 schematically depicts a vehicle moving on a roadway and including a roadway 

6 skeleton estimation constructed in accordance with the invention; 

7 FIG. 2 depicts a graph that schematically depicts a model of a roadway skeleton, useful in 
igS understanding one embodiment of the invention; 

■ ^9 FIG. 3 schematically depicts an image of a portion of a roadway, useful in understanding 

=f£0 outputs performed by the roadway skeleton estimation system; and 

M 

FIGS. 4 and 5 depict flow charts depicting operations performed by the roadway skeleton 

.12 estimation system in estimating the skeleton of the roadway, FIG. 4 depicting operations in 

j|3 connection with a model in which the roadway is modeled as a circular arc and FIG. 5 depicting 

* +4 operations in connection with a model in which the roadway is modeled as a parabolic arc. 

s i 5 Detailed Description of an Illustrative Embodiment 

1 6 FIG. 1 schematically depicts a vehicle 1 0 moving on a roadway 1 1 and including a roadway 

17 skeleton estimation system 12 constructed in accordance with the invention. The vehicle 10 may 

18 be any kind of vehicle 10 that may move on the roadway 11, including, but not limited to 

19 automobiles, trucks, buses and the like. The roadway skeleton estimation system 12 includes a 

20 camera 1 3 and a processor 1 4. The camera 1 3 is mounted on the vehicle 1 0 and is preferably pointed 

21 in a forward direction, that is, in the direction in which the vehicle would normally move, to record 

22 successive images as the vehicle moves over the roadway. Preferably as the camera 1 3 records each 

23 image, it will provide the image to the processor 14. The processor 14, in turn, will process 
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1 information that it obtains from the successive images, possibly along with other information, such 

2 as information from the vehicle's speedometer (not separately shown) to estimate a roadway skeleton 

3 representing a portion of the roadway 1 L ahead of the vehicle 10. The processor 14 may also be 

4 mounted in or on the vehicle 1 1 and may form part thereof. The roadway skeleton estimates 

5 generated by the processor 14 may be used for a number of things, including, but not limited to 

6 autonomous driving by the vehicle, providing assistance in collision avoidance, and the like. 

7 Operations performed by the processor 14 in estimating the roadway skeleton will be described in 

8 connection with the flow chart depicted in FIG. 3. 

9 Before proceeding further, it would be helpful to provide some background to the operations 
ip performed by the processor 14 in estimating the skeleton of the roadway 1 1 . This background will 
*Gi be described in connection with FIG. 2. Generally, the roadway is modeled as a circular arc parallel 

m 

i\2 to the XZ plane in three-dimensional space. The X (horizontal) and Y (vertical) axes of three- 

'1=3 dimensional space correspond to the "x" and "y" axes of the image plane of the images recorded by 

IP the camera 14, and the Z axis is orthogonal to the image plane. Preferably, the image plane will be 

i 5 the plane of the image after the image has been rectified to provide that the Z axis is parallel to the 

iSS plane o the roadway 1 1 ; the Stein I patent application describes a methodology for rectifying the 

|17 images to provide that the image plane will have a suitable orientation. 

;3 H 8 As noted above, the roadway is modeled as a circular arc, and, with reference to FIG. 2, that 

19 FIG. circular arc in the XZ plane representing the roadway, which will be referred to as the 

20 roadway's skeleton, is identified by reference numeral S. Two lane markings 20L and 20R are also 

21 shown as dashed lines on opposite sides of the skeleton S. The distances Rl and R2 between the 

22 lane markings 20L and 20R and the skeleton S may be the same, or they may differ. The skeleton 

23 S of the roadway is modeled as an arc extending from the vehicle 10, or more specifically from the 

24 image plane of the camera 14, which is deemed to be at point P=(0,0), the origin of the XZ plane. 

25 The center of the circular arc need not be along the X axis, and, indeed will generally be at an angle 

26 a thereto. The circular arc is parameterized by three components, namely 
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1 (i) the coordinates of the center (Xc, Yc) of the circular arc (the coordinates of the center will 

2 also be referred to as coordinates (a,b)); 

3 (ii) the angle a and the inverse of the radius 1/R, and 

4 (iii) for two locations Zl and Z2 along the Z axis, values XI and X2. In one embodiment, 

5 Zl and Z2 are three meters and thirty meters, respectively. 

6 For item (iii), value XI represents the horizontal distance between the Z axis, specifically the point 

7 with coordinates (0,Z1), and the point on skeleton S with coordinates (XI, Zl). A line LI is 

8 perpendicular to the line between point P=(0,0) and the center (Xc, Yc) of the circular arc comprising 
, :K 9 skeleton S intersecting the point P=(0,0). The value X2 represents the horizontal distance between 
CD the points on line LI and circular arc comprising skeleton S at coordinate Z2. In addition, it will 
fill be appreciated that the origin of the XY plane, point P=(0>0) also resides on the circular arc, so that 

;12 the coordinates of three points on the circular arc comprising the skeleton S, along with the 

'"•=.£ 

f Jj3 coordinates of the center of the circle that includes the circular arc, will be known. Parameterizing 

'14 the arc S in this manner will have several advantages. First, if the locations Zl and Z2 are relatively 

^5 far apart, depending on the radius R, the values XI and X2 may also be relatively far apart. Second, 

fl6 it would allow for use of a different road model, such as a parabolic arc, as will be described below. 

<F 

•f 7 Given a triplet (X 1 , X2,d), where "d" is the pitch (that is, the angle of the camera relative to 

IK 5 * 

18 the Z axis) of the camera, and an image x ¥ 9 it is desired to generate a warp of the image *F to a view 

19 in which the roadway is essentially a straight line. It is possible to warp the image onto the XZ 

20 plane, but instead the image is warped to the space R,p. Initially, given R, a and b, it should be 

21 recognized that 

22 R 2 = a 2 + b 2 (1). 

23 In addition 
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(2) 



1 

2 



and 



Z = (R+ Afl)sin(/? + fi 0 ) 



(3) 



5^4 where = tan" 



Kb) 



U 5 

&6 



ref- 



ill addition, since, for a point P(X,Y,Z) in three-dimensional space, the coordinates (x,y) of 
the projection of the point in the image are given by 



x = 



fx 



(4) 



8 and 
9 



y = 



z 



(5) 



10 
11 

12 



where X and Z are determined as above and Y is a function of the camera height Y h , the pitch "d" 
and the point's coordinate along the Z axis 



Y = -Y u + dZ 



(6). 
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In equation (6), Y h is a positive value and is subtracted from "dZ" since the surface of the roadway 
1 1 is below the camera 14. It will be appreciated that, at the point Z=0 directly below the camera, 
Y=Y h , as required. 

Since 

R 2 = (X x -af + (z x -bf (7) 



and 



R 2 



= (X 2 -af + (Z 2 -bf 



(8) 



and given the relation in equation (1), solving for "a" and "b" 



(x\ + z 2 )z, + (x 2 x + z])z 2 

a = 1 - 2 r 2 — ■ — 1 — v ; (9) 

2{X 2 Z X + X 2 Z 2 ) 



and 



X\ + Zf - 2aX, 

b = — 1 1 (10). 

2Z, 

After values for "a" and "b," the "X" and "Z" components of the center of the circular arc S in the 
XZ plane, the radius R can be determined using equation (1). 

Using the values for "a" and "b" determined using equations (9) and (10), the value for the 
radius R can be determined using equation (1). In addition, using equations (4) through (6) and the 
coordinates (x,y) of each point in the image *F, the coordinates (X,Y,Z) of points in three- 
dimensional space that are projected onto the image can be determined. Using the coordinates 
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1 (X,Y,Z), and the values for the radius R and p 0 , the image can be warped from the (x,y) space to a 

2 AR,P space using equations (2) and (3). 

3 As noted above, the image ¥ in rectangular ((x,y)) coordinates can be warped to AR, P space. 

4 At this point, it is desired to determine the range and resolution of those parameters. In one 

5 embodiment, it is desired to have the resolution on the order of one-tenth meter by one-tenth meter. 

6 Accordingly, it is desired to determine Ap such that ApR is on the order of a predetermined length, 

7 which, in one embodiment, is one-tenth meter. 

8 The range is determined as follows. As noted above, warping the image of the roadway to 

9 R,p space effectively provides a warped image V F I in which the roadway is straight, not curved. An 
B) illustrative image is depicted in FIG. 3. As shown in FIG. 3, roadway 30 is divided into two regions, 

including a near region 31 and a more distant region 32. In one embodiment, if the vertical 
'p coordinates of the picture elements, or "pixels," that are subtended by the roadway in FIG. 3 extends 
from - 1 20 to - 1 0, the vertical coordinates of the near region 3 1 extend from - 1 20 to -20. In relating 
ifA the image in Ap,R space to the physical roadway in three-dimensional coordinates, it should be noted 
: JL5 that the bottom line of the warped image 4 /t maps to a line of width AX at distance Z, where 

I* 6 fY 

,p Z = — (11) 

h y 

1 7 and, if, as shown in FIG. 3, the width of the image, with the horizontal coordinates extending from 

18 -160 to 160 pixels, is 320 

!9 AxZ AxfY hxY 320X 

AX = —— = - 1 — = = (12) 

f yf y -120 

20 The range of R, which is taken to be AR, is 3 AX. 

21 In determining the range of P, it should be noted that Z raax , the maximum value of Z for the 

22 roadway, is 



m 
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Z. 



(13). 



max 



-10 



Since 



sin(/? max + A))* = Z - 



max 



(14), 



the value of P max is 



= - A 0 + sin 



-l 




(15). 



max 



V R J 



With this background, operations performed by processor 1 4 will be described in connection 
with the flow chart in FIG. 4. Generally, the operations proceed in two phases. In the first phase, 
the processor 14 performs a rough alignment using a straight road model. In that operation, the areas 
of the image V F I are detected that appear to belong to road direction indicators, such as lane markings, 
tire marks, edges and so forth. In the second phase, the rough alignment generated during the first 
series of steps is used, along with information in the image including both regions 3 1 and 32 to 
determine a higher-order model. In that operation, only features found in the distant region 32 that 
are extensions of features found in the near region 31 are utilized, which will ensure that non- 
roadway features, such as automobiles, that may be found in the distant region 32 will be ignored. 

Accordingly, and with reference to FIG. 4, the processor 14 initially receives an image 
from the camera 14 (step 200). The image *F is a projection of points in rectangular three- 
dimensional coordinates (X,Y,Z). After receiving the image the processor 14 performs the first 
series of steps in connection with the near region 3 1 (FIG. 3) of the roadway to generate a first-order 
model of the roadway. Generally, for each of a predetermined number of selected values for X„ 
keeping predetermined values of the other parameters X 2 and radius R constant, the processor 1 4 will 
generate a warped image 4", and generates a value for a cost function for each warped image The 
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1 value of the cost function will represent a measure of the number of vertical features that are present 

2 in the warped image Since, for the warped image V P I for which the roadway appears to be 

3 straightest, the number of vertical features that are present in the warped image 4" will be largest, 

4 the processor 14 can select the value of the parameter X! that was used in generating the warped 

5 image associated with the largest cost function value as the appropriate value for parameter X v 

6 Mores specifically, and with further reference to FIG. 4, after the processor 14 receives the 

7 image it will select one, "i-th," of the selected values for parameter X, (step 201) and using the 

8 selected value and the predetermined values for parameters X 2 and R, generate a warped image ¥V, 

9 in which the image *F is warped to the R,P space (step 202). Thereafter, the processor 14 will 
XQ generate the cost function, as follows. The processor 14 initially generates a derivative image dY' 
fi along the horizontal ("x") coordinate of the image x ¥\ the derivative representing the rate of change 
ltj2 of image brightness, or intensity, as a function of the horizontal coordinate (step 203). Thereafter, 
Jf3 to reduce the possibility that one extremely bright object might dominate, the processor 14 applies 
Ili4 a non-linear function, such as a binary threshold or sigmoid function, to the derivative image $¥\ 
T5 thereby to generate a normalized derivative image n(d x P l ) (step 204). The processor 14, for each 
%6 column of pixels in the normalized derivative image n^Y') generated in step 204, generates a sum 

Bps 

|i7 of the pixel values of the pixels in the respective column (step 205), generates, for each column, the 

*I£8 square of the sum generated for the column (step 206) and forms a sum of the squares generated in 

149 step 206 as the value of the cost function for the value of the parameter X 1 and the warped image V P I 

20 generated therewith (step 207). In addition, the processor 14 determines, for each column, whether 

2 1 the value of the square exceeds a selected threshold (step 208). 

22 Following step 208, the processor 14 will determine whether it has performed steps 202 

23 through 208 in connection with all of the selected values for parameter X x (step 209). If the 

24 processor 14 makes a negative determination in connection with step 209, it will return to step 202 

25 to select another value for parameter Xj and performs steps 203 through 209 in connection therewith. 

26 The processor 1 4 will perform steps 202 through 209 in connection with each of the selected 

27 values for parameter X x to generate cost values therefor. When the processor 14 determines in step 

-10- 
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1 209 that it has performed steps 202 through 209 in connection all of the selected values for parameter 

2 Xj, it will sequence to step 209 to identify, among the cost function values generated in step 207, 

3 the maximum cost function value (step 210). In addition, the parameter 1 4 will determine the value 

4 of the parameter X, and the warped image ¥' associated with the maximum cost function value 

5 identified in step 209 (step 211). At this point the processor 1 4 will have completed the first phase, 

6 with the warped image comprising the rough alignment. 

7 After completing the first phase, the processor 14 begins the second phase. In the second 

8 phase, the processor 14 performs operations similar to those described above in the first phase, 

9 except that 

f@) (a) it performs the operations in connection with pixels not only in the near region 31, but 

|H also in the far region 32, and 

=fi (b) it generates the cost functions only in connection with columns that were determined in 

fJUB step 208 to exceed the threshold 

IP 

!JL4 (step 212) to determine the values for the respective parameters. 

i|5 The road skeleton S can also be modeled as a parabolic arc. In this case, it will be assumed 

:|6 that the major axis of the parabolic arc is along the X axis in three-dimensional space, which, as 

147 noted above, corresponds to the horizontal, or x, axis of the image *F. In that case, the road skeleton 

1 8 will conform to the equation 

19 X = aZ 2 + bZ+c (16), 

20 where coefficients "a," "b" and "c" are constants. The camera 14 is selected to be at the point 

21 (X,Z)=(0,0) on the skeleton S, in which case constant "c" is equal to zero. Lane markings are 

22 essentially horizontal translations from the skeleton, and therefore they can be modeled using 

23 equations of the same form as equation (16), and with the same values for coefficients "a" and "b"as 

24 that in the equation for the skeleton S, but different values for coefficient "c." It will be appreciated 
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that the value for coefficient "c" in the equations for the lane markings will indicate the horizontal 
displacement for the lane markings from the skeleton S for any point Z. 

In the case of the parabolic arc model, the image ¥ can be warped to an image 4", in which 
the skeleton is straight, as follows. If (x,y) are the coordinates of a point in image ¥ that is a 
projection of a point with coordinates (X,Y,Z) in three-dimensional space 

fx 

x = ~r 



y = 



(17). 



If a point in the image is a projection of a point the roadway, 

Y = -Y h + dZ (18). 



where, as above (see equation (6)), Y h is the height of the camera off the roadway and "d" is the pitch 
angle of the camera 14 relative to the horizontal axis Z. In equation (18), Y h is a positive value, and 
is subtracted from "dZ" since the surface of the roadway 11 is below the camera 14. It will be 
appreciated that at the point Z=0 directly below the camera, Y=Y h , as required. If values for "f," the 
focal length of the camera 14, Y h and d are known, it is possible to transform the coordinates (x,y) 
of points in the image to the coordinates (X,Z) of points in three dimensional space, and vice 
versa. The values of the parameters f and Y h are known and the value of parameter d will be 
determined in the course of determining values for coefficients "a" and "b" for the parabolic arc 
model equation (16). Also, given the values of coefficients "a" and "b" in equation ( 1 6), an overhead 
view (X,Z) of the skeleton of the roadway, that is, a view in the XZ plane, can be warped to a warped 
view (C,Z) in which the skeleton S has the equation C=0. In that case, roadway markings, which, 
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1 as noted above, are also modeled as parabolic arcs, will have equations C=c, where "c" is the value 

2 of the constant V* in equation 16 for the respective roadway markings. 

3 Accordingly, it is desired to determine the values of coefficients "a" and "b" and the pitch 

4 angle "d" that, when used to warp the overhead view (X,Z) of the skeleton S to the overhead view 

5 (C,Z), will result in a skeleton S that is a straight line. In that operation, a cost function is defined 

6 for the view (C,Z) whose value will be a maximum for the correct values of "a," "b" and "d." As 

7 with the methodology described above in connection with FIG. 4, searching for the maximum of the 

8 cost function's value is performed in two general phases, with a near region being used in the first 

9 phase, and a more extended region being used in the second phase. The value of the cost function 
W is determined as follows. Given assumed values for "a," "b" and "d," the image ¥ is warped to an 
41 image x ¥\ The image Y is an image of the roadway (and possibly other objects and features) in 
|fl which the skeleton of the roadway is, in an overhead (X,Z) view, a parabolic arc. On the other hand, 
:T3 image 4" is an image of the roadway (and possibly other objects and features) in which the skeleton 
Slj4 of the roadway may, in an overhead (C,Z) view, be a straight line, depending on the values for "a," 
;i5 "b" and "d" that were used in the warping. After the warped image ¥' has been generated, the 
4% warped image is projected onto the warped image's horizontal "x" axis by summing the pixel values 
1*7 in the respective columns. The derivative of the sums across the warped image's horizontal "x" axis 

is then determined, and a sum of the absolute values of the derivative across the warped image's 

§49 horizontal "x" axis is generated, with that sum being the value of the cost function. These operations 

20 are repeated for a number of assumed values for "a," "b" and "d" to generate a number of cost 

21 function values. 

22 The value of cost function value will be larger if the projection onto the warped image's 

23 horizontal projection is sharper, which will occur if there are more vertical features in the warped 

24 image ¥', and it would be possible to select the warped image Y' for which the cost function value 

25 is greatest as the warped image *F in which the skeleton S of the roadway appears as a straight line. 

26 If there were no objects in the image ¥, and hence warped image ¥', other than the roadway 1 1 , this 

27 would be correct. 

-13- 
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1 However, if there is clutter, such as other objects, in the image V, and hence warped image 

2 with strong edges, the value of the cost function tends to get dominated by the clutter, which may 

3 result in an improper warped image being selected. The two-phase methodology reduces the 

4 likelihood of this occurring. In the first phase, the image *F is warped to provide warped images ¥' 

5 using a straight road model, that is, with value of coefficient "a" set to zero and the values of "b" and 

6 "d" may be zero or non-zero. In each warped image 4", areas that appear to belong to road direction 

7 indicators, such as lane markings and tire tracks, as opposed to lines associated with other objects, 

8 such as cares, are then identified. In addition, during the first phase, a preliminary value for 

9 coefficient "b" and a value for pitch "d" are determined. 

ig Thereafter, in the second phase, image *F is again warped, to generate warped images 

=P using selected values for coefficient "a" and the preliminary value for coefficient "b" and the 

5f I value for pitch "d" that were determined during the first phase. Using areas vertically higher in the 

s l5 respective warped images, which represent areas in three-dimensional space (X, Y,Z) that are further 

Ijlt from the vehicle 10, the cost function values are generated, and an assessment as to the warped 

15 image for which the skeleton S most approximates a straight line. The value "a/ 1 that was used 

life in generating the warped image for which the skeleton S most approximates a straight line is 

1I7 selected as the value for coefficient "a" for the model represented by equation (16). The value pitch 

;|8 "d" identified in the first phase is selected as the pitch "d." The value of coefficient "b" value for the 

"T9 model represented by equation (16) corresponds to the value b which, as the coefficient of the linear 

20 term in the parabolic model for which the value is identified as the coefficient "a" in the 

2 1 parabolic model, most approximates the straight line model, using the preliminary value for "b," that 

22 was developed during the first phase. 

23 More specifically, and with reference to the flow chart in FIG. 5, after the processor 14 

24 receives an image Y (step 250), it will, in the first phase, select initial guess for the values for pitch 

25 "d" and coefficient "b" (step 251) and, using those values, warp the image Y (step 252) thereby to 

26 generate a warped image The processor 14 then, for each column of pixels in the warped image 

27 x ¥\ sums the pixel values for the pixels in the near region 31 in the column (step 253), generates 
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1 values representing the derivative in the horizontal direction (step 254) and generates values that 

2 correspond to the absolute value of the derivative (step 255). It will be appreciated that portions of 

3 the warped image 4" that contain, for example, edges or lane markings will produce peaks in the 

4 absolute values generated in step 255, since the edges and lane markings will correspond to fast 

5 transitions from dark-to-light or light- to-dark regions of the images. The processor 14 then 

6 thresholds the absolute values generated in step 255 (step 256), that is, for any absolute value that 

7 is less than a predetermined threshold value, the processor 14 sets the absolute value to zero. For 

8 the absolute values that are above the threshold, the processor 1 4 then finds local maxima (step 257). 

9 The local maxima, which will be identified as peaks "Pj" typically correspond to lane markings in 
1 0 the warped image although some peaks Pj may be caused by a strong edge that is not part of the 
% roadway, such as an automobile a short distance ahead of the vehicle 1 0. 

|| After identifying the peaks p i5 the processor 1 4 searches for values for the coefficient "b" and 

;fij pitch "d" using a search methodology for each of the peaks Pj (step 258). In that operation, in each 

Ilk of a series of iterations "j" Q=l J), the processor 14 selects a value dj for pitch d. In each iteration 

;i = 5 "j," the processor 14, in each of a series of iterations "k" (k=l,...), selects a value b k and performs 

%> steps 252 through 257 in connection therewith to find values V for which the respective peaks p ; are 

1*7 maximum. After completing the iterations "j" the set of points (dj,b k ! ) in a (d,b) plane that are 

•|8 associated with each peak p { form a line 1^ The lines l { that are associated with features along the 

If 9 roadway, such as lane markings and tire tracks, will intersect at or near a point (d^bj in the (d,b) 

20 plane, and the processor 14 will identify that point and determine the appropriate values for 

2 1 coefficient "b" and pitch "d" as b x and d^ respectively. If there are any obj ects that are not associated 

22 with features of the roadway 1 1 , such as automobiles, in the near region 3 1 that are associated with 

23 peaks p { , lines l { that are associated therewith will generally not be near point (d x ,b x ) and will be 

24 ignored. In addition, it will be appreciated that, since the lines \ { that are at or near point (d^bj are 

25 associated with features along the roadway, and the processor 14 can readily determine which 

26 columns in the warped image V P' contain those features. This, along with the values of coefficient 

27 "b" and pitch "d" will be used in the second phase. 
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1 In the second phase, the processor 14 determines a value for coefficient "a" using regions of 

2 the image including both the near region 3 1 and the far region 32, generally emphasizing columns 

3 that are associated with or near the subset of peaks ft that it identified as being associated with 

4 features of the roadway. In the second phase, the processor 14 initially selects a plurality of values 

5 aj (step 260) and, for each value % generates an adjusted value b t so that the line X - b x Z + c 

6 (where b x corresponds to b x determined in the first phase) best approximates the curve 

7 X = a ( Z + b t Z + c with the value of "c" being set to zero (reference equation 16) over a range 

8 of "Z" as determined in the first phase (step 261). Thereafter, the processor 14 selects a value of "a/' 

9 (step 262) and warps the original image *F to generate a warped image using the values "a/' and 

13 £ 

M "b x " as coefficients "a" and "6/ and value as the pitch "d" (step 263). The processor 14 then 

f ljl projects a portion of the warped image *F", specifically the portion in the far region 32, onto the 

"fe warped image's horizontal axis by summing the pixel values of the pixels in each column in that 

fJB region (step 264) and then generates the derivative of the sums along the horizontal axis (step 265). 

it . £ E 

T4 For columns that are associated with peaks p^ that the processor 1 4 had determined in the first phase 

1 3P5 were associated with features of the roadway 1 1 , such as lane markings and tire tracks, the processor 

f 16 14 will generate the absolute value of the derivative (step 266) and generate a value corresponding 

Jf 7 to the sum of the absolute values (step 267). It will be appreciated that, by performing steps 266 and 

1+8 267 in connection only with columns that were determined in the first phase to be associated with 

1 9 features of the roadway 1 1 , the processor 14 will minimize contributions due to features that are not 

20 associated with the roadway, such as automobiles, which may otherwise unduly influence the result. 

21 After performing steps 262 through 267 for the value "a/ 1 that was selected in step 262, the 

22 processor 14 will determine whether it has selected all of the values "a/ 1 that were selected in step 

23 260 (step 268), and, if not, return to step 262 to select another value "a/' and perform steps 263 

24 through 267 in connection therewith. The processor will perform steps 262 through 267 through a 

25 plurality of iterations until it has generated the sum of the absolute value of the derivative for all of 

26 the values "a/ 1 that were selected in step 260. When the processor 14 determines in step 268 that it 
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1 has selected all of the values "aj" that were selected in 260, it will have generated a sum of the 

2 absolute value of the derivative for all of the values "aj," in which case it will sequence to step 269. 

3 In step 269, the processor identifies the value for which the sum is the largest (step 269). The 

4 correct parameters for the skeleton S are, as the value of coefficient "a," the value ^ that was selected 

5 in step 269, as the value of coefficient "b," the value of b that was generated for that value in 

6 step 261, and, as the pitch "d," the value generated in the first phase. 

7 The invention provides a number of advantages. In particular, the invention provides a 

8 system for estimating the skeleton S of a roadway 1 1 for some distance ahead of a vehicle. Two 

9 specific methodologies are described, one methodology using a model in which the roadway is 
ftp modeled as a circular arc, and the other methodology using a model in which the roadway is 
Jffl modeled as a parabolic arc, in both methodologies requiring only one image, although it will be 
US appreciated that the system can make use of a combination of these methodologies, and/or other 
4J3 methodologies. Determining the skeleton of a roadway for some distance ahead of a vehicle can be 
] |4 useful in connection with autonomous or assisted driving of the vehicle. 

|1|5 It will be appreciated that a system in accordance with the invention can be constructed in 

^ r 6 whole or in part from special purpose hardware or a general purpose computer system, or any 

,p7 combination thereof, any portion of which may be controlled by a suitable program. Any program 

I J8 may in whole or in part comprise part of or be stored on the system in a conventional manner, or it 

19 may in whole or in part be provided in to the system over a network or other mechanism for 

20 transferring information in a conventional manner. In addition, it will be appreciated that the system 

2 1 may be operated and/or otherwise controlled by means of information provided by an operator using 

22 operator input elements (not shown) which may be connected directly to the system or which may 

23 transfer the information to the system over a network or other mechanism for transferring 

24 information in a conventional manner. 

25 The foregoing description has been limited to a specific embodiment of this invention. It will 

26 be apparent, however, that various variations and modifications may be made to the invention, with 

27 the attainment of some or all of the advantages of the invention. It is the object of the appended 
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claims to cover these and such other variations and modifications as come within the true spirit and 
scope of the invention. 

What is claimed as new and desired to be secured by Letters Patent of the United States is: 
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